[#192624] `multiple_es_nodes` by afharo · Pull Request #193899 · elastic/kibana

afharo · 2024-09-24T16:38:48Z

Summary

Related #192624.

ES complained because the archive was created with 8.0.0, but ES 9.0.0 requires the datastore to be upgraded to 8.16.0.
Then, it complained because the index from the archives was created on 7.13.0, and required reindexing.

The following steps have been followed to update the archives:

Start both nodes of ES 8.16 with the affected data-archive on separate terminals:
1. Node 01: yarn es snapshot --version=8.16.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01
2. Node 02: yarn es snapshot --version=8.16.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02

After ES is ready (without starting Kibana), reindex the index .kibana_7.13.0_001

Retrieve the settings from the original index via curl -L 'http://localhost:9200/.kibana_7.13.0_001' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''
Create the target index with those settings:

curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
"mappings": {
    "properties": {
        "bar": {
            "properties": {
                "status": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        },
        "coreMigrationVersion": {
            "type": "keyword"
        },
        "foo": {
            "properties": {
                "status": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        },
        "migrationVersion": {
            "dynamic": "true",
            "properties": {
                "bar": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "foo": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        },
        "namespace": {
            "type": "keyword"
        },
        "namespaces": {
            "type": "keyword"
        },
        "originId": {
            "type": "keyword"
        },
        "references": {
            "type": "nested",
            "properties": {
                "id": {
                    "type": "keyword"
                },
                "name": {
                    "type": "keyword"
                },
                "type": {
                    "type": "keyword"
                }
            }
        },
        "type": {
            "type": "keyword"
        },
        "updated_at": {
            "type": "date"
        }
    }
},
"settings": {
    "index": {
        "hidden": "true",
        "number_of_shards": "1",
        "number_of_replicas": "0"
    }
}
}'

Reindex the content: curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_001" }, "dest": { "index": ".kibana_7.13.0_002" } }'
Remove the old index and recreate the aliases

curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
  "actions": [
    { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana_7.13.0_001" } },
    { "remove_index": {"index": ".kibana_7.13.0_001" } },
    { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana_7.13.0" } },
    { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana" } }
  ]
}'

Stop both ES nodes.

Compress both archives

cd .es/node01/8.16.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data  -x "*/\.*"
cd ../../../
cd .es/node02/8.16.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data  -x "*/\.*"
cd ../../../

Run the tests to confirm that the issue is fixed: yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2024-09-24T16:38:51Z

Pinging @elastic/kibana-core (Team:Core)

afharo · 2024-09-24T16:39:10Z

    ...(nodes.length > 1
-      ? ['discovery.type=zen', `cluster.initial_master_nodes=${nodes.map((n) => n.name).join(',')}`]
+      ? [
+          'discovery.type=multi-node',


zen is no longer supported

jbudz · 2024-09-24T18:46:42Z

@@ -96,7 +96,7 @@ function createRoot({ logFileName, hosts }: RootConfig) {
 }

 // Failing 9.0 version update: https://github.com/elastic/kibana/issues/192624


comment can be removed

Addressed in a225053 (#193899)

jbudz · 2024-09-24T18:48:04Z

Do we know why the file size increased? mostly curious

#193741 (review)

I'll use ditto, sorry!

Addressed in 6a699b5 (#193899)

Reverted in 97b4937 (#193899). We noticed that the documents were gone 😨

I re-ran it locally and it included them now 🤷
However, somehow, the .zip generated via ditto were even larger than the ones generated with zip, so I stuck to the existing files 😇

This reverts commit 6a699b5.

gsoldevila

LGTM, thanks!

kibana-ci · 2024-09-25T11:30:31Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 97b4937

Failed CI Steps

Jest Tests #11

Test Failures

[job] [logs] Jest Tests #11 / Configuration button renders the tooltip correctly when hovering the button

Metrics [docs]

✅ unchanged

History

💚 Build #236735 succeeded 1c09fe2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afharo

jbudz

packages/kbn-test/src/es/test_es_cluster.ts

(cherry picked from commit 66f2027)

kibanamachine · 2024-09-25T13:22:11Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[#192624] `multiple_es_nodes` (#193899)](#193899)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>

## Summary Addresses #167676 Addresses #158318 Addresses #163254 Addresses #163255 #### Fix for `multiple_es_nodes.test.ts` Inspired on #193899 1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals: 1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 2. After ES is ready (without starting Kibana), reindex the index `.kibana_7.13.0_002` 1. Retrieve the settings from the original index via `curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''` 2. Create the target index with those settings: ```shell curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "mappings": { "properties": { "bar": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "coreMigrationVersion": { "type": "keyword" }, "foo": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "migrationVersion": { "dynamic": "true", "properties": { "bar": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "foo": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "settings": { "index": { "hidden": "true", "number_of_shards": "1", "number_of_replicas": "0" } } }' ``` 3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'` 4. Remove the old index and recreate the aliases ```shell curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "actions": [ { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } }, { "remove_index": {"index": ".kibana_7.13.0_002" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } } ] }' ``` 3. Stop both ES nodes. 4. Compress both archives ```shell cd .es/node01/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*" cd ../../../ cd .es/node02/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*" cd ../../../ ``` 5. Run the tests to confirm that the issue is fixed: `yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts` #### Fix for `incompatible_cluster_routing_allocation.test.ts` Inspired on #193741 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts ``` #### Fix for `read_batch_size.test.ts` Inspired on #193899 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts ```

…6641) ## Summary Addresses elastic#167676 Addresses elastic#158318 Addresses elastic#163254 Addresses elastic#163255 #### Fix for `multiple_es_nodes.test.ts` Inspired on elastic#193899 1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals: 1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 2. After ES is ready (without starting Kibana), reindex the index `.kibana_7.13.0_002` 1. Retrieve the settings from the original index via `curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''` 2. Create the target index with those settings: ```shell curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "mappings": { "properties": { "bar": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "coreMigrationVersion": { "type": "keyword" }, "foo": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "migrationVersion": { "dynamic": "true", "properties": { "bar": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "foo": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "settings": { "index": { "hidden": "true", "number_of_shards": "1", "number_of_replicas": "0" } } }' ``` 3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'` 4. Remove the old index and recreate the aliases ```shell curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "actions": [ { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } }, { "remove_index": {"index": ".kibana_7.13.0_002" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } } ] }' ``` 3. Stop both ES nodes. 4. Compress both archives ```shell cd .es/node01/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*" cd ../../../ cd .es/node02/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*" cd ../../../ ``` 5. Run the tests to confirm that the issue is fixed: `yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts` #### Fix for `incompatible_cluster_routing_allocation.test.ts` Inspired on elastic#193741 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts ``` #### Fix for `read_batch_size.test.ts` Inspired on elastic#193899 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts ``` (cherry picked from commit 3d254c2) # Conflicts: # src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

[elastic#192624] multiple_es_nodes

1c09fe2

afharo added chore Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor labels Sep 24, 2024

afharo self-assigned this Sep 24, 2024

afharo requested review from a team as code owners September 24, 2024 16:38

afharo commented Sep 24, 2024

View reviewed changes

jbudz reviewed Sep 24, 2024

View reviewed changes

afharo added 3 commits September 25, 2024 11:07

Remove comment

a225053

Recompress with ditto

6a699b5

Revert "Recompress with ditto"

97b4937

This reverts commit 6a699b5.

gsoldevila approved these changes Sep 25, 2024

View reviewed changes

afharo enabled auto-merge (squash) September 25, 2024 11:33

jbudz approved these changes Sep 25, 2024

View reviewed changes

afharo merged commit 66f2027 into elastic:main Sep 25, 2024

afharo deleted the 192624-multiple_es_nodes branch September 25, 2024 13:17

kibanamachine added the v9.0.0 label Sep 25, 2024

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 25, 2024

[elastic#192624] multiple_es_nodes (elastic#193899)

5755ea7

(cherry picked from commit 66f2027)

kibanamachine mentioned this pull request Sep 25, 2024

[8.x] [#192624] `multiple_es_nodes` (#193899) #194001

Merged

kibanamachine added the v8.16.0 label Sep 25, 2024

gsoldevila mentioned this pull request Oct 17, 2024

Update archives to 8.17.0 binary data format, unskip tests #196641

Merged

		@@ -96,7 +96,7 @@ function createRoot({ logFileName, hosts }: RootConfig) {
		}

		// Failing 9.0 version update: https://github.com/elastic/kibana/issues/192624

Conversation

afharo commented Sep 24, 2024 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

For maintainers

Uh oh!

elasticmachine commented Sep 24, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsoldevila left a comment

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Sep 25, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

jbudz left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Sep 25, 2024

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

afharo commented Sep 24, 2024 •

edited by kibanamachine

Loading